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Executive Summary 

This document will analyze two examples of the management-oriented approach to program 
evaluation. The first example explores the mathematics program in the Ohio County School District. The 
second example explores the entire school program at Ohio County High School. These program 
evaluations were both conducted to aid administrators and teachers in improving the curriculum across 
the entire district (as in the case of the mathematics audit) or at a specific school (as in the case of the 
scholastic review). To help the reader easily navigate this document, it will be divided into four sections. 
Sections one and two provide executive summary, literature, analysis, and summary information for each 
example. The third section contains a comparison of the two methods and explains how both fit the 
management-oriented model of program evaluation. The final section will summarize the findings. 

The management-oriented approach is very systematic in its approach to program evaluation. The 
systematic approach, which will be evidenced in the following pages, contributes to school accountability. 
Accountability is a reigning force in the American educational system today. Educational stakeholders are 
continually looking for ways to improve educational programs. No Child Left Behind and state mandates 
on schools have grave consequences for schools who fail to meet the established accountability marks. 

The ultimate goal for elementary and secondary institutions is to produce a well-educated 
citizenry capable of performing at high levels no matter what path may lie ahead. Federal and state 
accountability on schools has created a sense of need for all schools. Schools must be willing to evaluate 
programs in a structured and systematic way to ensure student achievement is continuous and 
progressing. The management-oriented models seem to provide the structure and organization for such 



program evaluations. 
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Section One 

Executive Summary for the Mathematics Audit 

Section one will provide a brief overview of the mathematics audit conducted in the Ohio County 
Schools between October 3 and October 5, 2005. The mathematics audit was conducted by Andersen, 
Baldwin, McWilliam Associates (ABMA) from Pocatello, Idaho. The ABMA report details how data was 
collected (through document reviews, personal interviews, teacher and administrator surveys, and focus 
group discussions among the teachers, administrators, parents and students at all grade levels in Ohio 
County, Kentucky), provides commendations for the school system, and provides findings to guide future 
decisions. Generated at the request of district-level administration and intended to provide building-level 
administrators and site based councils information to improve the mathematics programs within each 
school, this formative evaluation report details findings and recommendations for the areas of curriculum 
alignment, instruction strategies, and the administration of mathematics program throughout the district. 

The mathematics audit report contains a needs assessment evaluation of the mathematics 
curriculum and gives stakeholders suggestions for modifying the program to better meet the needs of the 
students. To my knowledge this was the first and only mathematics audit ever conducted in Ohio County 
Schools. It should be noted that the stakeholders within the district have incorporated many of the finding 
and conclusions. The incorporation of the auditors’ recommendations has resulted in an increase in 



student achievement as reflected on state and federal assessments. 
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Literature Review 

The schools that comprise the Ohio County School District are located in a small, rural county 
approximately one hundred miles southwest of Louisville, KY. The community is agriculture based with 
a few manufacturing companies. The school system, which serves approximately 3000 students in grades 
K through 12, has six elementary schools, one middle school, one high school and one alternative 
learning center which also serves as a Department of Juvenile Justice Day Treatment Center. The high 
school also allows students to take courses at the on-site advanced technical center, which is affiliated 
with the state’s community and technical college system. 

Mathematics instruction occurs at every grade level and was assessed for state and federal 
accountability puiposes at grades 5, 8, and 1 1 during the 2004-05 school year. Data gathered by 
Andersen, Baldwin, and McWilliam Associates reflected 2005 state testing data. The data provided 
showed 55% of the district’s fifth graders met the required proficiency level, 35% of the district’s eighth 
graders met the required proficiency level and 36% of the district’s eleventh graders met the required 
proficiency level. 

ABMA’s audit was divided into three parts: examination of data, on-site visits with the schools, 
and round-table discussions with teachers and administrators (Andersen, Baldwin, McWilliam Associates, 
2005, 5). The conclusions drawn by ABMA were provided so that decision makers could modify 
curriculum, instruction, and administration practices within the district. For the sake of brevity, this 
section will focus on only two of the findings and recommendations of the audit team. Table 1 lists these 



findings and recommendations. 
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Table 1 

Selected findings and recommendations 



Curriculum 



Finding: Elementary school curriculum maps were not aligned. Gaps existed between the 
various curriculum maps across the district. Many curriculum maps simply 
followed the table of contents of the textbook used. 

Recommendations: All curriculum maps within the district should be aligned to provide 
teachers with a prioritization of the content. This prioritization should 

not necessarily reflect a t extbook’s t a ble of contents. 

Curriculum 



Finding: Teachers were not using consistent vocabulary. Several teachers used 

inappropriate vocabulary in class, and students were often confused by the 
various terminology. 

Recommendations: Teachers should work collectively to generate vocabulary that reflects 
the content. This vocabulary should be clearly defined, and all teachers 
should incorporate t he co mmon term inology. 



As aforesaid, Ohio County has six elementary schools spread throughout the county. Placement 
in a specific school is determined by the home address of the family. If a family moves to a different area 
of the county, the children may attend a different elementary school. A change in school placement could 
have meant a student would miss mathematics content because curriculum maps were not consistent 
among all elementary schools. Individual schools had adopted curriculum maps that were used within 
their respective buildings, but no consideration had been given to the fact that a child might transfer 
between one or more schools during the course of a school year. Consistency among the curriculum maps 
was a key recommendation of ABMA. 

Content vocabulary is another important component of mathematics education. Consistent use of 
mathematics terminology contributes to student achievement. ABMA auditors observed that teachers used 
inaccurate terminology or modified definitions that failed to encompass the complete definition of the 
term. A rational number, for example, was defined by one teacher as “any number that can be written as a 
fraction” (Andersen, Baldwin, McWilliam Associates, 2005). The auditors determined this definition was 
deviated considerably from the more accurate definition of a rational number (that is, the ratio of two 
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integers). The auditing team recommended teachers use consistent and accurate terminology so students 
would not be as confused. 

Analysis 

Teachers and administrators discussed the findings, and each person worked to ensure the 
findings and recommendations were implemented. The findings enabled teachers to recognize the need 
for improvement and gave administrators clear guidelines for providing teachers with professional 
development opportunities. Administrators allowed teachers release time to work in collaborative 
groupings that transcended individual school boundaries. These focus groups met to align curriculum, 
generate vocabulary and discuss calculator usage at all grade levels. Elementary school teachers and 
principals across the district came together and adopted a single textbook and curriculum map. High 
school administrators and site based council members worked to rearrange teacher placement within the 
building so that all high school teachers would be departmentalized. 

The revised curriculum maps were implemented at all grade levels at the beginning of the 2006- 
07 school year - . Teachers incorporated key vocabulary and increased instructional time given to open 
response questions. Elementary school teachers dedicated more instructional time to the teaching of 
mathematics. Middle school administrators and teachers increased course rigor and utilized calculators at 
a more appropriate stage in each course. As a result of these modifications to the curriculum and 
instruction components of the mathematics program, state assessment results from the spring 2007 
assessment reflected an increase in the percentage of students achieving the defined level of proficiency. 
At the elementary school level, 63% of the students achieved proficiency as compared to 45% that had 
reached this mark two testing cycles earlier. At the middle school level, 63% of the students achieved 
proficiency as compared to 35% that had reached this mark two testing cycles earlier. At the high school 
level, 38% of the students achieved proficiency as compared to 36% that had reached this mark two 
testing cycles earlier. Obviously, the high school increase was minimal; however, the assessment of the 
high school instruction covers much more material than could have been “caught up” in one school year - . 
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Five years later, teachers and administrators still reflect on the report findings and look for ways 
to continue to improve instructional practices. The most recent test data (2009) shows a continued 
increase in student proficiency at all levels; high school data shows a 9% increase over the 2007 mark. 

The teachers and administrators credit this increase to the continued implementation and refinement of the 
recommendations given in the mathematics audit. The mathematics audit has become a continuous 
internal audit conducted among the stakeholders in the Ohio County Schools. 

Summary 

Ohio County Schools continuously work to improve student achievement and provide new 
teachers a means to immerse themselves in the progressive school culture that has resulted from the 
mathematics audit. The teachers that were present during the mathematics audit have championed the 
mathematics audit and work with new teachers to ensure that the impetus continues. Since teachers and 
administrators have such strong buy-in with the mathematics audit and its findings, there seems little need 
to seek outside evaluation of the program at this time. 

As with any education program, there is always room for growth. The mathematics department at 
the high school is working to ensure special populations are meeting annual yearly progress as measured 
by No Child Left Behind. This was not an issue specifically addressed in the findings of the audit team, 
but the high school’s implementation of the recommendations has aided the school in being more 
cognizant of the needs of the individual student. This increased awareness has led the school to seek 
programs to address specific learning outcomes. For example, the high school mathematics department 
has worked closely with the school council to secure funds for Carnegie Cognitive Tutor. Carnegie, a 
mathematics tutorial software, is designed to help strengthen student understanding of algebraic and 
geometric concepts. The high school council has also worked to secure Zoom Algebra, a calculator-based 
application, which helps lower-achieving students experience greater success with basic algebraic skills. 
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Section Two 

Executive Summary 

This section will analyze the Ohio County High School Scholastic Review conducted in 2002. 
Distinguished educators from the Kentucky Department of Education, led by Pearl Jean Hughes, visited 
Ohio County High School during the period of January 28 through February 1, 2002, to conduct the 
scholastic review. The team reviewed school portfolio and profile documents, conducted classroom 
observations, and interviewed teachers, students, parents, and administrators in both formal and informal 
settings. Ms. Hughes and her team used the Standards and Indicators for School Improvement as their 
primary tool of assessment, but also incorporated results from the pre-administered Effective Schools 
Survey as appropriate. 

The findings and recommendations of the scholastic review report were organized under three 
major headings: Academic Performance, Learning Environment and Efficiency. Each of these major 
headings had three standards that were assessed by the evaluation team. The intent of this 
review/evaluation was to guide the administrative staff, school council and faculty toward making OCHS 
a higher-achieving school. 

Many of the findings and recommendations of this report reference the instructional leadership of 
the school principal. It should be noted that the school principal retired at the end of the 2001-02 school 
year, prior to the release of the scholastic review report. The findings and recommendations of this report 
have been implemented by the current administrative staff and school council. In terms of student 
achievement and school accountability, several of the recommendations of the review team have led to 
program improvement, and all core content areas have had increases in student achievement as gauged by 



state assessment tools. 
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Literature Review 

Ohio County High School serves approximately 1 150 students in grades 9 through 12. The school 
employs approximately 75 certified staff members having various levels of experience. At the time of the 
scholastic review, the administrative staff included Principal Hayward Dean Minton and Assistant 
Principals Critt Hunter, Angela Alexander, and Greg Decker. 

As aforementioned, the review team used the Standards and Indicators for School Improvement 
(SISI) as the basis for the observations and interviews they conducted. These indicators “define the 
elements of whole school improvement that schools can put into effect at the elementary, middle and high 
school levels in order to produce desired learning results” (Kentucky Department of Education, 2009). 
SISI provides nine standards organized under three major categories. Academic Performance, the first 
category, incorporates a review of curriculum documents and information obtained through interviews 
with teachers, students, parents, and administrators. Academic Performance seeks to address “curriculum, 
classroom evaluation/assessment and instruction” (KDE Office of Leadership and School Improvement, 
2002). Learning Environment, the second category, incorporates a review of school council policies and 
practices, information obtained through interviews with teachers, students, parents and administrators and 
seeks to address “school culture; student, family, and community support; and professional growth, 
development and evaluation” (KDE Office of Leadership and School Improvement, 2002). Efficiency, the 
final category, incorporates a review of council policies and by-laws, curriculum documents, and 
information obtained through interviews with teachers, parents, and administrators; this category seeks to 
help the school “address leadership, school structure and resources, and comprehensive and effective 
planning” (KDE Office of Leadership and School Improvement, 2002). 

Each of the nine standards is subdivided into specific indicators that have been shown to 
contribute to school success. Based on best practices, these indicators are rated by the observation team 
on a one to four performance scale. According to the Kentucky Department of Education’s SISI School 
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Level Performance Indicators (2008), an Evaluation Category 1 rating reflects “little or no development 
and implementation,” an Evaluation Category 2 rating reflects “limited development or partial 
implementation,” an Evaluation Category 3 rating reflects “fully functioning and operational level of 
development and implementation,” and an Evaluation Category 4 rating reflects “exemplary level of 
development and implementation” (Kentucky Department of Education, 2008). Table 2 provides a 
summary of the ratings for each of the nine standards of education as observed during this visit to OCHS. 

Table 2 

Summary of Standards and Evaluation Categories for 2002 Scholastic Review of Ohio County High 
School 



Standard 

(Category) 


Evaluation Category (EC) 


Curriculum 


EC 1 - 1 indicator (14%) 


(Academic Performance) 


EC 2-6 indicators (86%) 


Classroom Evaluation/Assessment 
(Academic Performance) 


EC 2-8 indicators (100%) 


Instruction 


EC 1-1 indicator (13%) 


(Academic Performance) 


EC 2-7 indicators (87%) 


School Culture 


EC 1 - 1 indicator (9%) 


(Learning Environment) 


EC 2-9 indicators (82%) 
EC 3 - 1 indicator (9%) 


Student, Family and Community Support (Learning 


EC 2 - 4 indicators (80%) 


Environment) 


EC 3-1 indicator (20%) 


Professional Growth, Development, and Evaluation 


EC 1 - 8 indicators (67%) 


(Learning Environment) 


EC 2-3 indicators (25%) 
EC 3-1 indicator (8%) 


Leadership 


EC 1-6 indicators (55%) 


(Efficiency) 


EC 2 - 4 indicators (36%) 
EC 3 - 1 indicator (9%) 


School Organization and Fiscal Resources 


EC 1-2 indicators (20%) 


(Efficiency) 


EC 2-7 indicators (70%) 
EC 3-1 indicator (10%) 


Comprehensive and Effective Planning 


EC 1-2 indicators (13%) 


(Efficiency) 


EC 2-13 indicators (81%) 
EC 3-1 indicator (6%) 
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For purposes of brevity, this document will include a review of only one indicator from each of 
the three major categories. Information about the selected indicators that will be reviewed is presented in 
Table 3 (adapted from 2002 Scholastic Review). 



Table 3 

Selected indicators (adapted from 2002 Scholastic Review) 



Instruction indicator 3.1e 

Indicator: There is evidence that teachers incorporate the use of technology in their classrooms. 

Findings: Teachers have access to technology in the classroom, but their use of 
technology is limited to “college prep, career and technical classes.” 

Recommendations: OCF1S Council should develop a policy for the implementation of 

technology in all classes; administrators should monitor teacher use of 
technology as part of the evaluation process. 

Professional Growth , Development and Evaluation 6.2f 

Indicator: Leadership uses the evaluation process to provide teachers with the follow-up and support 
to change behavior and instructional practice. 

Findings: Administrators “do not use the evaluation process to provide support or follow- 
up for professional development.” 

Recommendations: School leaders should engage in professional development activities 
that “help build leadership capacity within the school to sustain long- 

range, contin uo us organizational i mprovem ent.” 

Efficiency 7.1c 

Indicator: There is evidence that all administrators have a growth plan focused on the development 
of effective leadership skills. 

Findings: Administrator growth plans “are not focused on leadership skills.” 

Recommendations: School principal should work “with the superintendent to develop 

goals and should enroll in the regional Kentucky Leadership Academy 
to develop and enhance instructional leadership skills.” 



Analysis 

By the time the official report from the Office of Leadership and School Improvement had been 
received, H. Dean Minton and Critt Hunter had retired. John A. Stofer, the newly appointed school 
principal, and his administrative staff worked diligently to address the numerous lower ratings received 
on the report. Many of the teachers in the building had been expecting a negative report, but were 
encouraged by the rapid changes Mr. Stofer brought. 
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The administrative staff of the school charted a new direction for the school. Shortly after the 
report was issued, the OCHS school council adopted the current mission statement for the school. That 
mission is “Stick to the focus of proficiency in teaching and learning.” The change in leadership seemed 
to bring a stronger sense of pride to the school and reinvigorated the entire faculty. Mr. Stofer, who had 
no previous administrative experience, has worked to build a collegial relationship with all faculty, and he 
has worked to ensure student success is the entire school’s number one priority. 

Changes initiated included the adoption of council policies to address technology inclusion in all 
areas of the curriculum, the adoption of professional development activities that build leadership skills for 
all administrators and faculty, and the appropriate use of the evaluation tool to gauge teacher 
effectiveness. The council policy on technology implementation has evolved into a School Improvement 
Plan component. This component details how all teachers should incorporate technology into the 
classroom and describes how specific departments use technology to promote instruction. The school 
district conducted renovations at the high school during 2008, and during that renovation ACTIV board 
technologies and projectors were installed in all classrooms. Computer lab space has more than doubled 
in the past two years so that teachers have more access to technology for their students. These 
improvements have encouraged teachers to embrace the technology that is available to improve student 
achievement. 

Administrators can easily evaluate teacher technology via the teacher’s formal and informal 
observations. The scholastic review has aided administrators in becoming more cognizant of the faculty’s 
professional development needs with regard to technology. At least six hours of professional development 
opportunities are planned each year to help teachers address growth areas in technology. 

In addition to the technology professional development offered annually, administrators conduct a 
needs assessment to determine what other professional development opportunities should be offered to 
the faculty. Administrators also participate in professional development opportunities. As evidenced by 
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the findings and recommendations aforementioned, the administrative staff was deemed ineffective in its 
use of the Ohio County School District’s evaluation process to appropriately identify teachers needing 
assistance. As a means of correcting this negative rating, administrators have attended numerous 
leadership workshops to learn how to closely follow the established evaluation plan. Administrators have 
increased the number of formal and informal visits to classrooms and have been retrained in the 
evaluation of teachers. New teachers are monitored on a more frequent basis; new teachers are also 
assigned resource teachers (or mentors) to increase their awareness of student achievement needs. These 
procedures have allowed administrators to have a better understanding of the professional needs of the 
entire faculty. 

Summary 

The program evaluation conducted by the Kentucky Department of Education’s Office of 
Leadership and Improvement is most closely aligned with the management-oriented approach. 
Specifically, this evaluation appears to be closely aligned with Daniel Stufflebeam’s Context, Input, 
Process, Product (CIPP) Evaluation Model (Stufflebeam, 2002). Stufflebeam’s model is useful in 
answering the question, “Is this program succeeding?” 

The model’ s first component, context, seeks to find the needs of the organization. In the case of 
OCHS, the needs are clearly aligned with the indicators/standards of highly effective schools. The 
model’s second component, input, seeks to have the evaluator review current documents and gain input 
from all stakeholders. The scholastic review clearly describes how information from stakeholders was 
gathered by the evaluation team and how this information was processed through a triangulation process. 
The triangulation process helps ensure one evaluator does not allow his/her opinion/bias to enter the 
situation; it seeks to find out if information gathered from one source can be corroborated with 



information others are receiving. 




Program Evaluation 14 



The process evaluation component of Stufflebeam’s model allows the evaluator to observe, 
document, and monitor the program firsthand. One-on-one interviews, classroom observations and round- 
table discussions with other evaluators help create valuable information and anecdotal evidence that can 
be used to create the report. 

The scholastic review conducted at Ohio County High School yielded numerous areas for 
improvement. Most, if not all, of these areas have been addressed by the current administrative staff. As 
alluded to in the Executive Summary, scholastic reviews are conducted to help the school improve 
practices which, in turn, help boost student achievement. Student achievement has risen at OCHS each of 
the past nine school years. Ohio County High School has transitioned into a data-driven school that is 
more technologically equipped and prepared to meet the needs of the students it serves. 

Section 3 

Comparison of the two examples 

The evaluations presented in sections two and three contain characteristics consistent with 
management-oriented program evaluation. This approach is rooted in a systematic approach and contains 
strict guidelines for the evaluator(s) to follow. Goals of the management-oriented approach include 
meeting the “information needs” of stakeholders (Crawford, 2006; Hogan, 2007). By meeting the 
informational needs of the stakeholders, the management-oriented approach allows decision makers to 
have a comprehensive view of the program. A comprehensive program evaluation helps identify and 
strengthen the program. 

As discussed in section two, Daniel Stufflebeam authored a checklist for the management- 
oriented approach. His framework is based on the need of the organization to answer questions about the 
program’s success. Stufflebeam states that program evaluation should follow the organized path of 
determining the context of the evaluation, gathering information for informed decision making, 
processing gathered information, and producing the final report of findings. 
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Marvin Alkin authored a framework similar to Stufflebeam’s. Alkin’s UCLA model details five 
areas a program evaluator should consider. These areas are systems assessment, program planning, 
program implementation, program improvement, and program certification (Key, 2002). Alkin’s UCLA 
model is more of a monitoring protocol for continued evaluation, but can be used for an initial program 
evaluation. 

In the case of the mathematics audit, district-level administrators sought to determine if the 
mathematics program was adequate in meeting the needs of the students in the Ohio County School 
District. External auditors provided the district administration the outside view that internal evaluators 
often cannot provide. The evaluation team conducted a thorough evaluation of the district’s mathematics 
program through its document review and interview sessions with stakeholders. This procedure is 
consistent with the protocol outlined in Stufflebeam’s CIPP and Alkin’s UCLA models. 

The scholastic review contained information about the high school’s complete educational 
program. Scholastic reviews arc usually conducted by educational leaders from outside the school 
district. These reviews arc not typically solicited. External evaluators provide unbiased findings and 
best-practice recommendations to help school decision makers improve the overall school program. 

The Ohio County High School principal and SBDM council utilized the data provided by both 
reports as a means to improve instructional decision making. Alkin’s UCLA model calls for a continued 
review of the evaluation document to ensure the goals and objectives of the program arc continuing to be 
met. The council refers to these documents frequently as it reviews accountability components and plans 
for departmental needs. Specific data from the mathematics audit and generalizations from the scholastic 
audit have benefited the council in determining a direction for the mathematics program to pursue. The 
pursuit of this refined vision has led to increased student accountability as evidenced earlier. 

In addition to council action, teachers have more closely monitored teaching practices. Teachers 
have actively sought activities, projects, and technology resources that link to the findings of both reports. 
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For example, a high school mathematics teacher adapted a video project he found on the Internet. This 
project had students explore the relationship between area and volume of an object if the object were 
scaled to a different size. The students identified an ordinary object and created a scale model by 
shrinking or enlarging to reflect a new object. Next, the students created a video about their object and 
explained the mathematics behind the enlargement and reduction. One group took a Golden Grahams 
cereal box and increased it to a 4.2 times larger box. The movie they created was based on a cave man 
that ravaged a town in search of food. The new Golden Grahams (toasted bread) satiated his hunger. The 
teacher’s increased awareness of the need to implement technology and increase student vocabulary may 
not have occurred had the scholastic review and mathematics audit not noted a deficiency in these areas. 

Section Four 

Summary 

Effective programs are critical to school success. Without program evaluations being used as a 
monitoring tool, programs can quickly become ineffective. Schools must continually maintain a healthy 
balance between reflection and vision to move the school forward. Schools in Ohio County arc striving to 
make maintain a healthy balance. This fact is evident in the examples provided in this document. Both 
the scholastic review and mathematics audit have contributed to the sustainability and subsequent growth 
of the program areas that were evaluated. 

The management-oriented approach, as presented by Stufflebeam and Alkin, has been used 
extensively in the Ohio County Schools in the past eight years. The Stufflebeam and Alkin models are 
very complementary of the leadership style of the current administration. Ohio County School District 
administrators have considerable data about specific programs, but often have limited knowledge of the 
specific content areas. The Stufflebeam CIPP and Alkin UCLA models allow administrators to ask the 
questions, but also provides them the flexibility to secure support from independent experts in the field to 



answer the questions. 
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Evidence provided in sections two and three of this document contributes to the viability of 
successful school programs. Without a concerted effort to follow through with the recommendations of 
the experts in the field, program failure would be inevitable. School program evaluations are a necessity 
and the Ohio County School District has proven that necessity and follow-through leads to student 
achievement. 
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